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Program description 1 



Waterford Early Reading Program ™ is a software-based cur- 
riculum for students in Kindergarten through second grade. The 
curriculum is designed to promote reading, writing, and typing, 
incorporating literacy skills such as letter mastery, language 
stories, spelling, basic writing skills, reading and listening 
development, and comprehension strategies. It can be used as a 



supplement to the regular reading curriculum. Program materials 
include classroom lessons and take-home materials in addition 
to the Waterford software. Waterford Early Reading Program™ 
offers pretest placement and posttest assessments, in addition 
to ongoing assessments throughout the program. 



Research One study of Waterford Early Reading Program™ met the What extent of evidence for Waterford Early Reading Program™ to 

Works Clearinghouse (WWC) evidence standards with reserva- be small for alphabetics and for comprehension. No studies 

tions. The single study included more than 70 Kindergarten that met WWC evidence standards with or without reservations 

students from six schools in Ohio. 2 The WWC considers the addressed fluency or general reading achievement. 



Effectiveness 



Waterford Early Reading Program™ was found to have potentially positive effects on alphabetics and no discernible effects on 



comprehension. 


Alphabetics 


Fluency 


Comprehension 


General reading 
achievement 


Rating of effectiveness 


Potentially positive effects 


na 


No discernible effects 


na 


Improvement index 3 


Average: +19 percentile points 
Range: -26 to +37 percentile 
points 


na 


Average: +4 percentile 
points 


na 

na = not applicable 



1. The descriptive information for this program was obtained from a publicly available document on the program’s web site ( http://www.pearsondiaital. 
com/pdfs/werp/researchsummarv-werp.pdf . downloaded April 2007). The WWC requests developers to review the program description sections for 
accuracy from their perspective. Further verification of the accuracy of the descriptive information for this program is beyond the scope of this review. 

2. The evidence presented in this report is based on available research. Findings and conclusions may change as new research becomes available. 

3. These numbers show the average and range of student-level improvement indices for all findings across the study. 
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Developer and contact 

Developed by Dustin Heuston, Ph.D., at the Waterford Institute, 
The Waterford Early Reading Program™ is distributed by 
Pearson Digital Learning. Address: 6710 East Camelback Road, 
Scottsdale, AZ 85251 . Email: pdlinfo@ pe arson.com . Web: http:// 
www.pearsondigital.com/waterford. Telephone: (888) 977-7900. 

Scope of use 

The program was initially developed in 1990, and the first level 
of the Waterford Early Reading Program ™ was launched in 
1995. According to the developer, it is currently used in more 
than 13,000 sites across the United States and serves 350,000 
students. 

Teaching 

Waterford Early Reading Program™ is divided into three levels, 
each designed for individualized, year-long instruction. The 
first level is taught in kindergarten and includes print concepts, 



phonological awareness, and letter recognition. The second level 
is taught in first grade and includes letter sounds, word recogni- 
tion, and beginning reading comprehension. The third level is 
taught in second grade and builds on levels one and two with 
an emphasis on content meaning of text and fluency in reading. 
Each level contains hundreds of songs and game-like activities 
with color graphics, digitized voices, and animation. 

The Waterford Early Reading Program™ curriculum includes 
the Waterford software, assessment materials, classroom les- 
sons, homework materials, and classroom posters, as well as 
student take-home books, CDs, and handouts. On-site training 
and online “webinars” are available for initial training in addition 
to a detailed teacher’s guide. 

Cost 

Program materials for the Waterford Early Reading Program™ 
cost approximately $300 per student. Cost of training is not 
available. 



Thirty-six studies reviewed by the WWC investigated the effects 
of Waterford Early Reading Program™. One study (Hecht & 
Close, 2002) was a quasi-experimental design that met WWC 
evidence standards with reservations. The remaining 35 studies 
did not meet WWC evidence screens. 

Hecht & Close (2002) included 42 students in four intervention 
schools and 34 students in two comparison schools. Students 
in the intervention schools received Waterford Early Reading 
Program ™ in addition to their regular curriculum. Students in 
the comparison schools received no supplement to their regular 
curriculum. 



Extent of evidence 

The WWC categorizes the extent of evidence in each domain as 
small or moderate to large (see the What Works Clearin gh o use 
E xtent of Evidence Ca t egoriza tion Scheme ). The extent of 
evidence takes into account the number of studies and the 
total sample size across the studies that met WWC evidence 
standards with or without reservations. 4 

The WWC considers the extent of evidence for Waterford 
Early Reading Program™ to be small for alphabetics and for 
comprehension. No studies that met WWC evidence standards 
with or without reservations addressed fluency or general read- 
ing achievement. 



4. The Extent of Evidence Categorization was developed to tell readers how much evidence was used to determine the intervention rating, focusing on the 
number and size of studies. Additional factors associated with a related concept, external validity, such as the students’ demographics and the types of 
settings in which studies took place, are not taken into account for the categorization. 
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Effectiveness 



The WWC found Waterford 
Early Reading Program ™ 
to have potentially positive 
effects on alphabetics 
and no discernible effects 
on comprehension 



WWC Intervention Report 



Findings 

The WWC review of interventions for beginning reading 
addresses student outcomes in four domains: alphabetics, 
fluency, comprehension, and general reading achievement. 5 
The study reported here included outcomes for alphabetics and 
comprehension. Within alphabetics, the studies reviewed cover 
four constructs: phonological awareness, letter knowledge, print 
knowledge, and phonics. 

Alphabetics. Hecht & Close (2002) examined nine student 
outcomes in the alphabetics domain: four phonological awareness 
outcomes (the Elision, Phonemic Blending, Phonemic Segmenting, 
and Sound Matching subtests of the Comprehensive Test of Pho- 
nological Processing (CTOPP)); one letter identification outcome 
(a letter name knowledge test); one print awareness outcome 
(Stones— Concepts About Print test); and three phonics outcomes 
(the Letter Sound Knowledge and Letter Word Identification 
subtests of the Woodcock-Johnson Tests of Achievement-Revised 
and the Spelling subtest (with phonemic representation scoring) 6 of 
the Wide Range Achievement Test (WRAT)). Hecht & Close (2002) 
reported statistically significant positive effects of the Waterford 



Early Reading Program™ for all nine outcomes. However, the 
WWC analysis found that none of these effects were statistically 
significant. The average effect size across all nine outcomes was 
large enough to be considered substantively important according 
to the WWC criteria (that is, an effect size of at least 0.25). 

Comprehension. The study authors examined one vocabulary 
development outcome (the Vocabulary subtest of Stanford-Binet, 
Fourth Edition) and reported no statistically significant effect. The 
effect size was neither statistically significant nor substantively 
important. 

Rating of effectiveness 

The WWC rates the effects of an intervention in a given outcome 
domain as positive, potentially positive, mixed, no discernible 
effects, potentially negative, or negative. The rating of effective- 
ness takes into account four factors: the quality of the research 
design, the statistical significance of the findings, 7 the size of 
the difference between participants in the intervention and the 
comparison conditions, and the consistency in findings across 
studies (see the WWC Intervention Rating Scheme). 



Improvement index 

The WWC computes an improvement index for each individual 
finding. In addition, within each outcome domain, the WWC 
computes an average improvement index for each study and 
an average improvement index across studies (see Technical 
Details of WWC-Conducted Computations). The improvement 
index represents the difference between the percentile rank 
of the average student in the intervention condition versus 
the percentile rank of the average student in the comparison 
condition. Unlike the rating of effectiveness, the improvement 



index is based entirely on the size of the effect, regardless of 
the statistical significance of the effect, the study design, or the 
analyses. The improvement index can take on values between 
-50 and +50, with positive numbers denoting results favorable to 
the intervention group. 

The average improvement index for alphabetics is +19 
percentile points for the single study, with a range of -26 to +37 
percentile points across findings. The improvement index for the 
single outcome in the comprehension domain is +4 percentile 
points. 



5. For definitions of the domains, see the Beginning Read ing Prot ocol . 

6. Spelling is not typically counted as a WWC Beginning Reading outcome, but this subtest used a qualitative scoring method. See Appendix A2.1 for 
details. 

7. The level of statistical significance was reported by the study authors or, where necessary, calculated by the WWC to correct for clustering within class- 
rooms or schools and for multiple comparisons. For an explanation, see the WWC Tutorial on Mismatch . See the Technical Details of WWC-Conducted 
Computations for the formulas the WWC used to calculate the statistical significance. In the case of Waterford Early Reading Program™, corrections for 
clustering and multiple comparisons were needed. 
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The WWC found Waterford 
Early Reading Program ™ 
to have potentially positive 
effects on alphabetics and 
no discernible effects on 
comprehension (continued) 



Summary 

The WWC reviewed 36 studies on Waterford Early Reading 
Program™. One study met WWC evidence standards with 
reservations; the remaining studies did not meet WWC evidence 



screens. Based on the one study, the WWC found potentially 
positive effects on alphabetics and no discernible effects on 
comprehension. The evidence presented in this report may 
change as new research emerges. 



References Met WWC evidence standards with reservations 

Hecht, S. A., & Close, L. (2002). Emergent literary skills and train- 
ing time uniquely predict variability in responses to phonemic 
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of Experimental Child Psychology, 82(2), 93-115. 

Additional citation for this study: 

Hecht, S. A. (2000). Waterford Early Reading program in Ohio: 
An evaluation. (Available from the Waterford Institute, Inc., 
55 West 900 South, Salt Lake City, UT 84101) 

Did not meet WWC evidence screens 
Alfaro, R. (1999). The technology-reading connection. Educa- 
tional Workshop, 56(6), 48-51. 8 
Canedo, M., Smolen, L., & Pollard, J. (2000). A study of the 
effectiveness of the Waterford Early Reading Program: 
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8. Does not use a strong causal design: this study did not use a comparison group. 

9. Complete data are not reported: the WWC could not compute effect sizes because complete study details were not reported. 

10. Does not use a strong causal design: there was only one intervention and one comparison unit, so the analysis could not separate the effects of the 
intervention from other factors. 

11 . Does not use a strong causal design: a historical cohort was used as the comparison group. WWC conventions allow for historical control cohorts in 
studies that have a broad unit of analysis (school or higher). This study analyzed at the student level and therefore does not fulfill the WWC requirement. 

12. Incomparable groups: this study was a quasi-experimental design that used achievement pretests but it did not establish that the comparison group 
was comparable to the treatment group prior to the start of the intervention. 
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For more information about specific studies and WWC calculations, please see the WWC Waterford Early 
Reading Program ™ Technical Appendices. 



16. Confound: this study included Waterford Early Reading Program™ but combined it with another intervention so the analysis could not separate the 
effects of the intervention from other factors. 

17. Complete data are not reported: the WWC could not evaluate the design because complete data were not reported. Attempts to contact the authors for 
more information were unsuccessful. 
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Appendix 



Appendix A1 


Study characteristics: Hecht & Close, 2002 (quasi-experimental design) 


Characteristic 


Description 


Study citation 


Hecht, S. A., & Close, L. (2002). Emergent Literacy Skills and Training Time Uniquely Predict Variability in Responses to Phonemic Awareness Training in Disadvantaged 
Kindergartners. Journal of Experimental Child Psychology, 82(2), 93-115. 

Additional source: 

Hecht, S. A. (2000). Research Compendium: The Waterford Early Reading Program. (Available from Waterford Institute, Inc., 55 West 900 South, Salt Lake City, UT 84101). 
(Study: Waterford Early Reading program in Ohio.) 


Participants 


The study began with 140 full-day, at-risk Kindergarten students who were randomly selected from six schools. Students from four schools who received the Waterford Early 
Reading Program™ were matched to students in two schools who did not receive the program. Students were pretested in the fall and posttested in the spring of the same 
school year. Because of mobility and absences, 64 students attrited from the study. The final analysis sample included 76 students. The mean age of students was five years 
and seven months. The majority of students were eligible to receive free/reduced lunch. 1 The majority of students in the schools came from low socio-economic status and 
African-American families. 


Setting 


The study took place in six inner city or rural public schools in Ohio. 


Intervention 


Students received the computer-assisted instruction of Waterford Early Reading Program™- Level One (WERP-1) during their normal classroom lessons for six months. The 
program focused on phonological awareness skills, letter knowledge, print concepts, and oral language skills. Students worked on the Waterford multimedia computer on their 
own for 15 minutes each session. A teacher management system was used to track daily time use. 


Comparison 


Students in the comparison group received their regular reading curriculum and were not exposed to the Waterford Early Reading Program™. 


Primary outcomes 
and measurement 


Nine outcomes were assessed in the alphabetics domain including the Comprehensive Test of Phonological Process (Phonemic Segmenting, Phonemic Blending, Elision, and 
Sound Matching subtests), the Woodcock-Johnson Tests of Achievement (the Letter Word Identification subtest), the Wide Range Achievement Test (the Spelling subtest with 
Phonemic Representation scoring), the Concepts About Print Test, the Letter Name Knowledge and Letter Sound Knowledge measures, and the Stanford-Binet: Fourth Edition 
Vocabulary subtest. The study also used a letter writing task from the Spelling subtest of the Wide Range Achievement Test, but this test was outside the domains specified by 
the Beginning Reading protocol (see Appendices A2.1-2.2 for more detailed descriptions of outcome measures). 


Teacher training 


Information about teacher training was not provided in the study. 



1 . The WWC received additional information on the analytic sample from the study authors. Baseline equivalence of the intervention and comparison group students remaining in the study was 
demonstrated by the authors. 
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Appendix A2.1 Outcome measures in the alphabetics domain 



Outcome measure 


Description 


Phonological awareness 

Comprehensive Test of 
Phonological Processing 
(CTOPP): Elision subtest 


A standardized measure of children's phonological awareness skills. Children were asked to say a word. Then children were asked what the word would be if a specific 
phoneme in the word were deleted. The remaining phonemes were used to form a word (as cited in Hecht, 2000). 


CTOPP: Phonemic 
Blending subtest 


A standardized measure of children’s phonemic synthesis skills. It includes four practice items and 15 test items consisting of two- to four-phoneme, one- and two-syllable 
words. This test measures the total number of words correctly spoken (as cited in Hecht & Close, 2002). 


CTOPP: Phonemic 
Segmenting subtest 


A standardized measure of children’s phonemic analysis skills. It includes three practice items and 15 test items consisting of two- to five-phoneme single-syllable words. This 
test measures the total number of words correctly pronounced one phoneme at a time (as cited in Hecht & Close, 2002). 


CTOPP: Sound 
Matching subtest 

Letter identification 


A standardized measure of children’s sound matching skills. Children were asked to pick which of three pictured words began with the same first sound as a target word (as 
cited in Hecht, 2000). 


Letter Name Knowledge 
Print awareness 


A researcher-developed measure designed to measure the total number of letter names correctly pronounced (as cited in Hecht & Close, 2002). 


Concepts About Print Test 
Phonics 


This 18-question test (Stones version) yielded one score reflecting students’ knowledge about print. The score is measured by the total number of correct items (as cited in 
Hecht & Close, 2002). 


Letter Sound Knowledge 


A researcher-developed measure designed to measure the total number of letter sounds correctly pronounced (as cited in Hecht & Close, 2002). 


Wide Range Achievement 
Test: Spelling subtest 
with phonemic 
representation scoring 


Students wrote 15 words as dictated by the test administrator. Scoring was based on Wilkinson’s method of giving partial credit for accuracy of phonemic representation (as 
cited in Hecht & Close, 2002). Students received between 0 and 6 points depending on how many and the placement of phonemes were represented by phonemically related 
or conventional letters in each written word. 


Woodcock-Johnson Tests of 
Achievement-Revised: Letter 
Word Identification subtest 


A standardized measure of children’s word reading. Children identified various letters of the alphabet as well as words, ranging from commonly used words to less familiar 
words of the English language (as cited in Hecht, 2000). 
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Appendix A2.2 Outcome measure in the comprehension domain 



Outcome measure 


Description 


Vocabulary 


Stanford-Binet (4th ed.): 
Vocabulary subtest 


A standardized measure to assess general cognitive ability and estimate general verbal IQ. The score is measured by the total number of correctly defined words (as cited in 
Hecht & Close, 2002). 
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Appendix A3.1 Summary of study findings included in the rating for the alphabetics domain by construct 1 



Authors’ findings from the study 
Mean outcome 









(standard deviation 2 ) 




WWC calculations 








Sample size 






Mean difference 3 




Statistical 






Study 


(schools/ 


Waterford 


Comparison 


(Waterford- 




significance 5 


Improvement 


Outcome measure 


sample 


students) 


group 


group 


comparison) 


Effect size 4 


(at a = 0.05) 


index 6 









Hecht & Close, 2002 (quasi-experimental design) 7 








Phonological awareness 

CTOPP: Elision subtest 


Kindergarten 


6/76 


4.71 

(3.45) 


2.82 

(2.39) 


1.89 


0.62 


ns 


+23 


CTOPP: Phonemic 
Blending subtest 


Kindergarten 


6/76 


9.53 

(5.55) 


4.24 

(5.08) 


5.29 


0.98 


ns 


+34 


CTOPP: Phonemic 
Segmenting subtest 


Kindergarten 


6/76 


7.58 

(7.05) 


1.53 

(2.84) 


6.05 


1.07 


ns 


+36 


CTOPP: Sound Matching subtest 


Kindergarten 


6/76 


10.91 

(4.71) 


6.27 

(4.89) 


4.64 


0.96 


ns 


+33 


Letter identification 

Letter Name Knowledge 


Kindergarten 


6/76 


21.58 

(4.43) 


24.65 

(4.14) 


-3.07 


-0.71 


ns 


-26 


Print awareness 

Concepts About Print Test 


Kindergarten 


6/76 


8.58 

(3.05) 


9.01 

(4.57) 


-0.43 


-0.11 


ns 


-4 


Phonics 

Letter Sound Knowledge 


Kindergarten 


6/76 


19.09 

(8.87) 


22.55 

(9.33) 


-3.46 


-0.38 


ns 


-15 


WRAT: Spelling subtest with 
phonemic representation scoring 


Kindergarten 


6/76 


25.57 

(19.67) 


8.09 

(7.79) 


17.48 


1.11 


ns 


+37 


Woodcock-Johnson Tests of 
Achievement-Revised: Letter 
Word Identification subtest 


Kindergarten 


6/76 


3.54 

(3.43) 


0.77 

(1.16) 


2.67 


0.99 


ns 


+34 


Domain average 8 for alphabetics 












0.50 


ns 


+19 



ns = not statistically significant 

1 . This appendix reports findings considered for the effectiveness rating and the average improvement indices for alphabetics. 
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Appendix A3.1 Summary of study findings included in the rating for the alphabetics domain by construct (continued) 



2. The standard deviation across all students in each group shows how dispersed the participants' outcomes are: a smaller standard deviation on a given measure would indicate that participants had more similar outcomes. The standard 
deviations for CTOPP Elision and Sound Matching subtests were received from the first author. 

3. Positive differences and effect sizes favor the intervention group; negative differences and effect sizes favor the comparison group. The intervention group mean in this table equals the comparison group mean plus the mean difference. 
The mean difference is calculated as the difference between gain scores and takes into account the pretest difference between the study groups. 

4. For an explanation of the effect size calculation, see Technical Details of WWC-Conducted Computations . 

5. Statistical significance is the probability that the difference between groups is a result of chance rather than a real difference between the groups. 

6. The improvement index represents the difference between the percentile rank of the average student in the intervention condition versus the percentile rank of the average student in the comparison condition. The improvement index 
can take on values between -50 and +50, with positive numbers denoting results favorable to the intervention group. 

7. The level of statistical significance was reported by the study authors or, where necessary, calculated by the WWC to correct for clustering within classrooms or schools and for multiple comparisons. For an explanation about the 
clustering correction, see the WWC Tutorial o n Mismatc h. See Technical Deta ils of WWC-C on ducted Computation s for the formulas the WWC used to calculate statistical significance. In the case of Hecht & Close (2002), corrections for 
clustering and multiple comparisons were needed, so the significance levels differ from those reported in the original study. 

8. This row provides the study average, which in this instance, is also the domain average. The WWC-computed domain average effect size is a simple average rounded to two decimal places. The domain improvement index is calculated 
from the average effect size. 



WWC Intervention Report Waterford Early Reading Program™ 



July 16, 2007 



11 




